Standards for Test Automation

By: Brian Tervo

Overview
Over the last five years, I’ve had the opportunity to work in a group who writes automated tests using retail Windows applications to ensure compatibility in future releases of Windows, and to find bugs in the operating system. During that time, we have transitioned from having no standards to having a strict set of guidelines to follow when writing new automated tests. We have seen many benefits as a result of developing standards. In this paper, I will talk about these three areas:

1. Why there is a need for standards in test automation
2. Things to consider when developing test automation standards
3. Methods to ensure everyone adopts and follows the standards once they’re written


Why there is a need for standards in test automation I will start with a hypothetical situation. Suppose a company has decided to hire a team of testers to write test automation. They start by hiring two testers, Sandy and Joe. Both of these testers are given a test tool in a shrink-wrapped box, and asked to start writing automated test cases. The testers work independently, learn about the test tool and develop their own library of routines. They start implementing some test cases with automation and begin to see some success.

About a month later, the company hires two more testers, James and Sue. The manager of the team asks Sandy to mentor James and Joe to mentor Sue. Sandy shows James her library of routines. James completely adopts Sandy’s framework, and eventually starts adding some of his own routines to Sandy’s library files. Likewise, Joe shows Sue his library of routines. Although Sue finds most of the routines in Joe’s library useful, there are some fundamental aspects of his framework that she does not like. After a couple of weeks, she makes copies of Joe’s library files and starts modifying them to her suit her coding style.

A few more weeks pass, and the need for more automated tests increases, so the company allocates funds to have two more testers to write automation. They transferred Stephanie from within the company to the automation team because she has recently attended some courses on how to write test automation, and wanted the opportunity to use what she had learned. After glancing through the library files others on the team have created, she realized that others on the team made some fundamental mistakes based on what she learned in her classes. As a result, she wrote yet another new set of library functions. Shortly after Stephanie joined the team, the company hired Bob who just graduated with a Computer Science Degree. Bob was a little disappointed with this job since he really wanted to be a developer and was upset that he had to resort to working in a test group. When Bob reviewed the library files from others on the team, he decided that none of the libraries was well organized or well structured. Bob created his own library of routines, which used linked lists, structures, pointers, and several other programming elements that the test tool was not designed to handle.

More months passed, and thousands of lines of test code were written. Since each tester was responsible for running the code they wrote on a regular basis, they found themselves spending their entire time running and triaging their tests, with no time left over to write new tests. To get the testers back to writing more automation, the company decided to move all of the tests to a lab and hire a lab manager to run the tests using a test harness.

Moving tests to a lab turned out to be a very expensive process. All of the libraries and several of the test scripts had to be modified so they could be run with the test harness. Several of the scripts had dependencies, such as screen resolution, environment variables, disk space, presence of certain drive letters, etc. that could not always be replicated in the lab environment. The dependencies varied from one library to the next. To move all of the tests to a single lab, many compromises had to be made, and each tester had to spend several weeks adapting their individual libraries to work in the lab.

The group finally got most of their automated tests running in a lab environment, shipped their product a couple months later, and started testing the next version of the product. Since most of the test cases for the old version of the product could still be run on the new version, the company chose to keep their automation team, and to continue writing more test cases for features being added to the new version. There were, however, a couple of major changes that happened shortly after the product shipped.

Bob accepted a job as a developer in another part of the company. The ownership of his code was distributed equally among the remaining members on the team. Unfortunately, nobody on the team was able to understand or Bob’s code because of its complexity. They each spent many hours trying to determine the causes for script failures, and eventually came to the conclusion that their time would be better spent rewriting Bob’s test cases using their own libraries. In effect, several months of Bob’s work was lost because he was the only one who could fully understand how his code worked.

They encountered another problem when the new version of the product added a new feature that broke a major function in the test tool. After pleading with program managers and developers to remove the feature from the product, they realized that was a losing battle. They were going to have to find a way to work around the issue in their test tool. Stephanie was the first to come up with a solution in her library, and she showed her new function to the group. The rest of the group was excited about her innovation, and tried to call her library function directly. However, there were problems with doing that. Stephanie’s function had dependencies on several of the other files in her library. If others on the team wanted to use her function, they had to include all of Stephanie’s library files. This led to additional problems when Stephanie’s libraries happened to use the same global variable names and function names as files in different libraries. As a result, nobody could use Stephanie’s code directly, and they ended up having to copy and paste her code into their own library files and maintain it separately.

I will end this story here. It illustrates some of the common problems associated with not having standards for test automation.

1. When global changes are required, everyone has to make the changes. In this story, this happened twice. The first time was when the team decided they needed to run their tests under a harness, and code had to be added to the libraries to support the harness. If they had a single library, they would only need to modify one library rather than several libraries. The second example was when Stephanie wanted to share code from her library, but couldn’t because of dependencies with her other library files. If they had a single library, Stephanie’s new function would have been immediately available to everyone in the group.

2. Tests are limited to run in the environment in which they were written. This first became an issue for this test team when they wanted to move their tests to a lab. They learned that their tests had dependencies, and that these dependencies varied from one library to the next. If they had a standard machine configuration, or took the time to find out which dependencies needed to be handled by the test automation, the transfer of tests to a lab environment could have gone much smoother.

3. Tests can only be debugged or maintained by the author. When Bob left the group, he left behind several months of work that could not be used, because nobody else in the group could understand how his code worked. If the team had standards on coding structure and style, everyone would be able to read each other’s tests, and Bob’s code would not have been lost when he left the team.

Things to consider when developing test automation standards

It’s important to look at the entire scope of your automation project when you create standards for test automation. If you overlook something while drafting the standards, it can be expensive to change standards after your team has created several thousand lines of code. Here is a list of some of the things you should consider when creating standards for test automation:

1. Test Tools. There are many automation test tools on the market. Some of them were designed for testing a specific class of applications. For example, some tools were written with the intent of testing GUI applications, and have functionality to manipulate objects commonly found in the UI of the operating system. Other tools were written to test Web-based applications and have built-in functions to manipulate objects commonly found on Web pages. Most test tools have both a scripting language and capture/playback tools. However, some are intended as scripting tools and to create tests by writing code, while others were designed to be primarily for capture/playback. Chances are you will use both the capture/playback and the scripting language features, regardless of which tool you choose. The capture/playback features of scripting tools can often be used to obtain code samples to learn more about the scripting language. But some amount of time is almost always required to tweak the recorded scripts to get them to run reliably and return viable pass/fail information during playback. All test tools have benefits and drawbacks, and it’s beyond the scope of this paper to discuss all the details of what. each test tool does or does not offer. For further information on how to select a test automation tool, I recommend reading “Making the Right Choice” and “Evaluating Tools” by Elisabeth Hendrickson, listed in the references section at the end of this paper.

2. Libraries. I recommend that each project have just one library and that there is no duplication of code or functionality in the library. For example, if there is a library file with that already contains several functions to create random numbers, all random number functions in the library should be in that file. If a tester needs some new feature for generating random numbers, either an existing function in the library should be modified in a way that will not break any script currently using it, or a new function should be added to the random number function library file. Having just one library of routines rather than several will save time in library maintenance.

3. Logging. Having a common logging format is fundamental for automation that can be understood and debugged by everyone on the team. Consider what the log files should look like. Should they be in plain text or HTML? Should information be logged to a database? Where will the log files be stored? What information should be logged? At a quick glance, will any tester on the team be able to review a log from a test, determine whether the test passed or failed, and if necessary triage a failure? Will people who are not familiar with the automation be able to read the log files and understand the results? Does the test tool have built in logging functionality? If so, will it meet your needs, or will it be necessary to write a custom logging engine?

4. Error Handling. What should the tests do when they encounter an error? Should they log pertinent information about the failure, attempt to clean up, and move on to the next test? Or, should they stop everything, leaving the software in the failure state until someone investigates the failure? Should there be different levels of error handling, where the error is handled differently based on the severity of the failure?

5. Environment. What is the default test environment? For example, will all tests run at only one screen resolution and color depth? Or, will the tests have to run with several different resolutions and color depths? If so, which ones? How many logical drives are on the test machine? Will all the machines have the same number of drives with the same letter assignments? Will all drives with the same letter have the same file system? How much free space will be on each drive? Which operating system(s) will the tests have to run on? Which language versions of the software will be supported? Will the application under test be installed before any tests begin? If so, where will it be installed? If not, how will each script detect that the application is not installed and install it? Can each individual script assume there will always be a printer installed on the test machine? Which environment settings will the scripts be responsible for setting? Which settings can the scripts optionally change, and must they be returned to the default setting if the script ends? What happens if the script stops running before it changes things back to the defaults? Are there library functions available to set and restore each environment setting?

6. Test Harnesses. What information should the harness provide the test script about the application and the environment? What information should the test script provide the harness? What additional code, if any is required for the tests to interface with a test harness? Can any of this code be included in a library so it’s transparent to each individual script?

Full article...

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/